Take the black level of RAW images from the optical black - #158

Split out of the draft #157, which keeps the parts that are still open. This PR holds only what is finished.

1. The black level of RAW images comes from the optical black

Problem. sum_col_val carries an additive offset that does not vanish when the LED dims, so the extinction coefficients saturate in dense smoke. The problem statement of #145 names three sources of it. One is that the black level LEDSA subtracts comes from the RAW metadata, which is a nominal value of the camera model and differs from the actual level of the sensor by a fraction of a count. Integrated over the search area, that fraction becomes a constant offset of the intensity of every LED.

Change. _black_level takes the black level from the mean of the optical black pixels of the image itself. These are masked pixels at the margin of the sensor, read out together with the image, so the estimate also follows the drift of the black level with the sensor temperature. raw.black_level_per_channel[channel] stays as the fallback for sensors whose margin is narrower than MIN_OPTICAL_BLACK_WIDTH (8 px). The package subtracts a black level in exactly one place, _read_img_array_from_raw_file, so the change reaches sum_col_val, bgsub_sum_col_val, sidebg_sum_col_val and bgsub2_sum_col_val alike. JPG and PNG have no black level and are untouched.

It is not an option, because nobody can reasonably prefer a nominal black level to a measured one, and because an option would turn the background correction into black level handling times spatial estimate, which the layout dispatch by column count cannot carry.

The results do change for RAW images, which breaks the "default results are unchanged" rule that #145 was merged under. That is the purpose of the change. The column layout does not show it, so the step 3 header now records the LEDSA version, as the headers of the analysis results already do.

2. The binary data are rewritten instead of appended

Two pre-existing problems next to #92, found while checking what part 1 breaks:

• create_binary_data wrote all_parameters.h5 with append=True and nothing removed the file of an earlier run, so every LED ended up in the table once per run and the analysis saw each image as often as the step was repeated. The loop always reads the files 1 to N from the start, so appending could only ever duplicate and was never able to extend the table with images it had not already read. The file is now removed and rewritten, and the replacement is reported.
• The column names are taken from the first result file of a channel and used for all of them, so result files of different width cannot be loaded together. That happens exactly when step 3 is run again for part of the images, which part 1 invites. Until now pandas stopped with Shape of passed values is (n, 12), indices imply (n, 10), which says nothing about the cause. The error now names both files, both counts and the re-run of the data extraction.

Backwards compatibility

• No column is added, no config option, no API change, so every step 3 layout is untouched and old config.ini and config_analysis.ini files run unchanged.
• The width a result file is compared against is len(columns) - 3, which holds for every branch of _get_column_names, the two legacy ones included (LEGACY_FIT_COLUMNS at 18 columns and _get_old_columns at 15 and 4), so the new check rejects no layout that loaded before.
• The extra , LEDSA x.y.z on the first header line is inert: read_table skips the comment lines and nothing parses the first one.
• Installing this alone changes no number. The values live in the step 3 result files, so they move when step 3 is run again, not when the version is installed. Re-running only the analysis on existing result files gives the old numbers.
• Delete analysis/channel*/all_parameters.h5 when re-running the analysis on data extracted with an older version. With part 2 that is no longer silent.

Tests

Unit tests pass, 191. New are test_black_level.py for the estimate and the fallback and six tests in test_file_handling.py for the two guards. The fake RAW image of the orientation tests gained the margin sizes, so it now covers the fallback as well.

Not covered: no RAW image is read by any unit test, so the optical black estimate itself still rests on the comparison against BCL on the Warsaw tests V008 and V009, camera 2. The acceptance tests were not run locally. The unit tests ran under numpy 2.2.6 here, outside the numpy ~= 1.26.4 of pyproject.toml, so the CI is the first run on the supported version.

What is not in here

The other half of the correction stays in the draft #157: dropping the clipping at 0, the ob_sum_col_val and ob_noise_sum_col_value columns, the SNR weighting and the image averaging. Dropping the clipping lets the sums go negative and so changes a contract that max_col_value, the fit and the postprocessing rely on, and it needs the nonlinear solver and the stacked analysis to be covered first.

🤖 Generated with Claude Code